Concerning the Bit Error Rate (BER) and channel capacity optimization in the data transfer process of Multi-Input Multi-Output (MIMO) system, a multi-user MIMO multi-hop relay system based on cooperative likelihood detection and sphere decoding was proposed. Firstly, it constructed a second-order cooperative MIMO relay system model to analyze the relay transmission process of channel data, as well as path loss and shadow fading. Secondly, the sphere decoding was used to derive the equivalent maximum likelihood rule for log-normal shadow fading detection. Finally, the maximum channel power harmonic mean selection policy was put forward, and an access link with smaller BER was chosen for user based on correlation link metric and maximum channel power threshold value, so as to improve the performance of multi-user MIMO system. Simulation results show that, compared with the multi-user MIMO multi-hop relay system based on mutual information maximization and the relay system based on decoding-forwarding and MIMO orthogonal Space Time Block Code (STBC), the average BER of the proposed system reduced by 27.4% and 32.6% respectively, and the average channel capacity increased by 9.5% and 12.7% respectively. The results demonstrate that the proposed system has a good effect in reducing BER and improving channel capacity.